Learning to Compress Ergodic Sources

We present an adaptive coding technique which is shown to achieve the optimal coding in the limit as the size of the text grows, while the data structures associated with the code only grow linearly with the text. The approach relies on Huffman codes which are generated relative to the context in which a particular character occurs. The Huffman codes themselves are inferred from the data that has already been seen. A key part of the paper involves showing that the loss per character incurred by the learning process tends to zero as the size of the text tends to infinity. This involves an analysis in an on-line learning framework bounding the cumulative loss, where loss is defined to be the excess code length. By using the Bayes prediction distribution and code the expected loss per character converges to zero at the best possible rate of O(log n/n). By allowing the length of contexts to grow in response to commonly occurring subsequences, the coding is efficient precisely where it needs to be, hence achieving a high compression rate at a relatively low overhead in terms of data structure storage. *Joint affiliation with Department of Mathematics, London School of Economics and Department of Computer Science, Royal Holloway, University of London 1068-0314/96$5.0001996IEEE 423 Proceedings of the 1996 Data Compression Conference (DCC) 1068-0314/96 $10.00 © 1996 IEEE